Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: R2Qiling with refactored memory and de-flatten plugin #1244

Open
wants to merge 15 commits into
base: dev
Choose a base branch
from

Conversation

chinggg
Copy link
Contributor

@chinggg chinggg commented Oct 13, 2022

Checklist

Which kind of PR do you create?

  • This PR only contains minor fixes.
  • This PR contains major feature update.
  • This PR introduces a new function/api for Qiling Framework.

Coding convention?

  • The new code conforms to Qiling Framework naming convention.
  • The imports are arranged properly.
  • Essential comments are added.
  • The reference of the new code is pointed out.

Extra tests?

  • No extra tests are needed for this PR.
  • I have added enough tests for this PR.
  • Tests will be added after some discussion and review.

Changelog?

  • This PR doesn't need to update Changelog.
  • Changelog will be updated after some proper review.
  • Changelog has been updated in my PR.

Target branch?

  • The target branch is dev branch.

One last thing


The draft PR #1213 refactors ql.mem by changing its internal implementation, which cannot get agreement from all developers. So I keep the existing code almost intact and move the promising but unstable feature into R2Qiling and R2Mem. Users can just replace Qiling with R2Qiling to enjoy and test the r2-powered Qiling.

@@ -166,7 +166,7 @@ def __perms_mapping(ps: int) -> str:

return ''.join(val if idx & ps else '-' for idx, val in perms_d.items())

def __process(lbound: int, ubound: int, perms: int, label: str, is_mmio: bool) -> Tuple[int, int, str, str, str]:
def __process(lbound: int, ubound: int, perms: int, label: str, is_mmio: bool, _data: bytearray = None) -> Tuple[int, int, str, str, str]:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this?

Copy link
Contributor Author

@chinggg chinggg Oct 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The wrapper R2Mem may call get_mapinfo then __process each entry in map_info, I add this optional argument since map_info in R2Mem has 6 elements instead of 5. I don't want to copy much code which only differs in # of parameters. Most importantly, it will not affect the behavior of existing code.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@elicn Do you approve these changes on the existing code?

@chinggg chinggg force-pushed the r2qiling branch 4 times, most recently from 33892d8 to becf881 Compare October 20, 2022 07:48
Copy link
Member

@elicn elicn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the effort you put in to this!
Please go over the code, clean and tidy it up, so it would be more intuitive to use out of the box.

examples/extensions/r2/deflat_r2.py Outdated Show resolved Hide resolved
examples/extensions/r2/deflat_r2.py Outdated Show resolved Hide resolved
examples/extensions/r2/hello_r2.py Show resolved Hide resolved
examples/extensions/r2/mem_r2.py Outdated Show resolved Hide resolved
examples/extensions/r2/mem_r2.py Outdated Show resolved Hide resolved
qiling/extensions/r2/__init__.py Outdated Show resolved Hide resolved
@xwings
Copy link
Member

xwings commented Nov 10, 2022

@wtdcode are we ready to merge this ?

@chinggg
Copy link
Contributor Author

chinggg commented Nov 11, 2022

Currently, the example code in hello_r2.py like r2.strings is broken since we pass RBuffer to r2 instead of the whole binary and r2 fails to load complete bininfo.

Documentation is also required. I will continue to revise it in December after I complete my graduate applications.

@xwings
Copy link
Member

xwings commented Nov 11, 2022

once this is merged and some minor bug fix. we can tag a release

@wtdcode
Copy link
Member

wtdcode commented Nov 12, 2022

No, unfortunately it's not ready to merge yet and probably needs a few more revise.

@chinggg
Copy link
Contributor Author

chinggg commented Dec 23, 2022

It seems #1293 has made some improvements to memory management functions like del_mapinfo. I will take a look and update this branch.

in addition to 'invalid' instruction
BUG: mips32 uc map 0x9000000 become 0x1000000
Add new class AnalOp and Operand
@Property r2.offset returns current offset

New methods of R2:
write(addr: int, bs: bytes)
get_fcn_at(addr: int) -> Function
get_bb_at(addr: int) -> BasicBlock
get_fcn_bbs(addr: int) -> List[BasicBlock]
dis(Function | BasicBlock) -> List[Instruction]

New attributes of R2Data:
__str__ can hexlify int
__contains__  makes range checking easier
@Property: start_ea, end_ea

Fix an error in @aaa decorator to accept args
example deflat_r2.py tests a x86 program compiled with OLLVM -mllvm -fla

see https://blog.quarkslab.com/deobfuscation-recovering-an-ollvm-protected-program.html
assert_mem_equal is only needed for R2Qiling
refactor r2._cmd() to allow optional r_core passed
@wrap_arg_addr makes function accept name/R2Data as addr and return same func when args is empty

rename: get_fcn_at -> get_fcn
rename: get_bb_at -> get_bb
r2.shell() can now be launched at an address (PC by default)
only difference: use an extra dict cmap to store ctype buf

map_info is the same as existing, no bytearray stored and updated
@chinggg
Copy link
Contributor Author

chinggg commented Dec 31, 2022

I have tried to keep the r2 extension compatible with existing code, now there is no extra record of bytearray in map_info, not sure if there is other problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants